{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating Deep Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this last section, we'll construct our networks from scratch.\n",
    "\n",
    "The target problem is again classification of Higgs Boson data.  \n",
    "\n",
    "Let's load BIDMat/BIDMach"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import BIDMat.{CMat,CSMat,DMat,Dict,IDict,Image,FMat,FND,GDMat,GMat,GIMat,GSDMat,GSMat,HMat,IMat,Mat,SMat,SBMat,SDMat}\n",
    "import BIDMat.MatFunctions._\n",
    "import BIDMat.SciFunctions._\n",
    "import BIDMat.Solvers._\n",
    "import BIDMat.JPlotting._\n",
    "import BIDMach.Learner\n",
    "import BIDMach.models.{FM,GLM,KMeans,KMeansw,ICA,LDA,LDAgibbs,Model,NMF,RandomForest,SFA,SVD}\n",
    "import BIDMach.networks.{Net}\n",
    "import BIDMach.datasources.{DataSource,MatSource,FileSource,SFileSource}\n",
    "import BIDMach.mixins.{CosineSim,Perplexity,Top,L1Regularizer,L2Regularizer}\n",
    "import BIDMach.updaters.{ADAGrad,Batch,BatchNorm,IncMult,IncNorm,Telescoping}\n",
    "import BIDMach.causal.{IPTW}\n",
    "\n",
    "Mat.checkMKL\n",
    "Mat.checkCUDA\n",
    "Mat.setInline\n",
    "if (Mat.hasCUDA > 0) GPUmem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And define the root directory for this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val dir = \"/code/BIDMach/data/uci/Higgs/parts/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constructing a deep network Learner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"Net\" class is the parent class for Deep networks. By defining a learner, we also configure a datasource, an optimization method, and possibly a regularizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "val (mm, opts) = Net.learner(dir+\"data%03d.fmat.lz4\", dir+\"label%03d.fmat.lz4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to define the network to run. First we set some options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "opts.hasBias = true;                    // Include additive bias in linear layers\n",
    "opts.links = iones(1,1);                // The link functions specify output loss, 1= logistic\n",
    "opts.nweight = 1e-4f                    // weight for normalization layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we define the network itself. We'll import a couple of classes that define convenience functions to generate the nodes in the network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import BIDMach.networks.layers.Node._\n",
    "import BIDMach.networks.layers.NodeSet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll define the network itself. Each layer is represented by a function of (one or more) input layers. \n",
    "Layers have optional arguments, specified in curried form (second group of parentheses). The layer types include:\n",
    "* input layer - mandatory as the first layer.\n",
    "* linear layer - takes an input, and an optional output dimension and bias\n",
    "* sigmoid layer - σ or \"sigmoid\" takes a single input\n",
    "* tanh layer - \"tanh\" with a single input\n",
    "* rectifying layer - \"rect\" with a single input (output = max(0,input))\n",
    "* softplus layer - \"softplus\" with a single input\n",
    "* normalization layer - takes an input and a weight parameter\n",
    "* output GLM layer - expects a \"links\" option with integer values which specify the type of link function, 1=logistic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "val in = input;                                                  // An input node\n",
    "val lin1 = linear(in)(outdim = 1000, hasBias = opts.hasBias);    // A linear layer\n",
    "val sig1 = σ(lin1)                                               // A sigmoid layer\n",
    "val norm1 = norm(sig1)(weight = opts.nweight)                    // A normalization layer\n",
    "val lin2 = linear(norm1)(outdim = 1, hasBias = opts.hasBias);    // A linear layer\n",
    "val out = glm(lin2)(irow(1))                                     // Output GLM layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we assemble the net by placing the elements in an array, and passing a NodeSet from them to the Learner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "val mynodes = Array(in, lin1, sig1, norm1, lin2, out);\n",
    "opts.nodeset = new NodeSet(mynodes.length, mynodes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning Options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here follow some tuning options "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "opts.nend = 10                         // The last file number in the datasource\n",
    "opts.npasses = 5                       // How many passes to make over the data \n",
    "opts.batchSize = 200                  // The minibatch size\n",
    "opts.evalStep = 511                    // Count of minibatch between eval steps\n",
    "\n",
    "opts.lrate = 0.01f;                    // Learning rate\n",
    "opts.texp = 0.4f;                      // Time exponent for ADAGRAD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You invoke the learner the same way as before. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mm.train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets extract the model and use it to predict labels on a held-out sample of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "val model = mm.model.asInstanceOf[Net]\n",
    "\n",
    "val ta = loadFMat(dir + \"data%03d.fmat.lz4\" format 10);\n",
    "val tc = loadFMat(dir + \"label%03d.fmat.lz4\" format 10);\n",
    "\n",
    "val (nn,nopts) = Net.predictor(model, ta);\n",
    "nopts.batchSize=10000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run the predictor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nn.predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate, we extract the predictions as a floating matrix, and then compute a ROC curve with them. The mean of this curve is the AUC (Area Under the Curve)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "val pc = FMat(nn.preds(0))\n",
    "val rc = roc(pc, tc, 1-tc, 1000);\n",
    "mean(rc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plot(rc)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Tune It!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try varying your nets design to see how accurate it can be. Feel free to write procedural code - i.e. generate your need using a loop and customize the layer sizes. What was your final accuracy ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "scala",
   "version": "2.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
